Point of clarification: in the end, was the issue about the size of a given key or was it the overall size of the document? I thought it seemed to be the former but turned out to be the latter, but I could be misremembering or behind. Worth nailing that down here for the record in case we revisit later.

CI shows that this test is failing:

I think this test (mine) was not the right approach. The query planner's internal details are slippery, and may depend on the number of records, perhaps the available system resources, the version of Postgres or SQLite...

I think it would be best to develop a separate test harness, which may or may not use pytest, to evaluate index usage. Like performance benchmarks, this is just a different category of test and does not sit well inside the unit test suite. I think I'd be in favor of just ripping this out for now, but I'm open to alternatives.

@jmaruland Your (C2QA) use case is one motivation for this fix. At your convenience, would you check out this PR branch locally, from @Kezzsim's fork, start a temp catalog server, and test that you can insert your data?

Point of clarification: in the end, was the issue about the size of a given key or was it the overall size of the document? I thought it seemed to be the former but turned out to be the latter, but I could be misremembering or behind. Worth nailing that down here for the record in case we revisit later.

Technically it isn't key related, it's overall metadata size related. Here's an example error from importing a bluesky run:

sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.ProgramLimitExceededError'>: index row size 2792 exceeds btree version 4 maximum 2704 for index "top_level_metadata"
DETAIL:  Index row references tuple (40,10) in relation "nodes".
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
Consider a function index of an MD5 hash of the value, or use full text indexing.
[SQL: INSERT INTO nodes (key, ancestors, structure_family, metadata, specs) VALUES ($1::VARCHAR, $2::JSONB, $3::structurefamily, $4::JSONB, $5::JSONB) RETURNING nodes.id, nodes.time_created, nodes.time_updated]
[parameters: ('b603b6ae-5cd8-43d7-b340-b7d8678ad7c4', '[]', 'container', '{"start":{"time":1527797320.2107823,"uid":"b603b6ae-5cd8-43d7-b340-b7d8678ad7c4","motors":["dwell_time_dwell_time","dcm_energy"],"XDI,Mono,name":"Si( ... (5836 characters truncated) ... mp":1527797320.2107823,"datetime":"2018-05-31T20:08:40.210782+00:00","plan_name":"scan_nd","stream_names":["primary"],"duration":178.91345500946045}}', '[]')]
(Background on this error at: https://sqlalche.me/e/20/dbapi)

I will remove the term keys from the PR title and description for clarity

Once the CI passing, we must remember to add a database migration that will drop the old index and replace it with the new one, on existing deployments. (I can pair-code this on Zoom, @Kezzsim; the process is not too complicated, but a little obscure.)